Vehicle Tracking and Speed Estimation from Video
Objective: To detect, track, and estimate the speed of vehicles from a video feed using computer vision techniques and YOLO-based object detection.
Methodology¶
1. Object Detection & Tracking
- Used the YOLO11n model to detect vehicles in each video frame.
- Implemented object tracking across frames to maintain vehicle IDs even if detections were temporarily lost.
2. Road Region Extraction
- Applied Canny edge detection and Hough Line Transform to identify road boundary lines.
- Incorporated constraints (expected line angles, approximate road height) and thresholding to filter out irrelevant lines.
- Created a trapezoidal mask representing the road region.
3. Bird’s-Eye View Transformation
- Used perspective transformation to map the trapezoidal road region into a rectangular bird’s-eye view.
- This simplifies vehicle motion analysis and distance measurement.
4. Vehicle Counting
- Defined a reference line across the road in the bird’s-eye view.
- Counted vehicles crossing the line in both directions.
5. Speed Estimation
- Estimated velocity by tracking vehicle displacement over time in the bird’s-eye view.
- Converted pixel displacement per frame into speed using frame rate and perspective scaling.
6. Result Visualization
- Combined all outputs into a single annotated video using OpenCV (cv2.VideoWriter).
- Overlaid:
- Detected vehicles with bounding boxes
- Road mask and bird’s-eye view
- Counting line and vehicle counts
- Estimated speed values
Import Libraries
¶!pip install -q ultralytics
!pip install -q openvino
!pip install -q 'lap>=0.5.12' 'pytubefix>=6.5.2'
#!pip install -q vidstab
import cv2
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
import numpy as np
from IPython.display import display, clear_output
Stabilize Video¶
Provided Video for this project video source has no fixed view point. The video can be stabilized using this:
from vidstab import VidStab
stabilizer = VidStab()
# Use XVID or MP4V for compressed output
try:
stabilizer.stabilize(
input_path='/kaggle/input/traffic-videos/4K Video of Highway Traffic - Nicholas Abraham-Raegan Martinez (720p h264).mp4',
output_path="stable_video.mp4",
border_type='black',
output_fourcc='mp4v',
)
except cv2.error:
pass # ignore destroyAllWindows error in headless env
#Re-encode with ffmpeg (much smaller file size, same quality):
!ffmpeg -i /kaggle/working/stable_video.mp4 -vcodec libx264 -crf 23 -preset slow stable_small.mp4
from ultralytics import YOLO
model = YOLO("yolo11n.pt")
# creates 'yolo11n_openvino_model/'
model.export(format="openvino",
half=True) # CPU Intel Xeon supports float16 inference
# Load the exported OpenVINO model
ov_model = YOLO("yolo11n_openvino_model/", task="detect")
Ultralytics 8.3.203 🚀 Python-3.11.13 torch-2.6.0+cu124 CPU (Intel Xeon CPU @ 2.20GHz) YOLO11n summary (fused): 100 layers, 2,616,248 parameters, 0 gradients, 6.5 GFLOPs PyTorch: starting from 'yolo11n.pt' with input shape (1, 3, 640, 640) BCHW and output shape(s) (1, 84, 8400) (5.4 MB) OpenVINO: starting export with openvino 2025.3.0-19807-44526285f24-releases/2025/3... OpenVINO: export success ✅ 5.2s, saved as 'yolo11n_openvino_model/' (5.4 MB) Export complete (5.7s) Results saved to /kaggle/working Predict: yolo predict task=detect model=yolo11n_openvino_model imgsz=640 half Validate: yolo val task=detect model=yolo11n_openvino_model imgsz=640 data=/usr/src/ultralytics/ultralytics/cfg/datasets/coco.yaml half Visualize: https://netron.app
The YOLO11n model tracks vehicle instances based on frame association. Occasionally, the vehicle cannot be seen in certain frames due to misidentification or obstruction by other objects. The YOLO tracking method features a tolerance frame window to prevent object misses. Additionally, there are frames where objects are tracked. These frames, without object location, can be filled with frames interpolated from information in nearby frames. In our case, with simple linear interpolation, the vehicles do not change direction.
import warnings
# ignore merge warnings
warnings.filterwarnings("ignore", category=RuntimeWarning)
def interpolate_boxes(ID, tracks_detected):
"""
track: dict mapping frame_idx -> (x1, y1, x2, y2)
returns: dict with interpolated boxes
"""
_temp = tracks_detected.query(f"ID == {ID}")[["Frame", "Box"]].sort_values(by="Frame").values
frames, track = list(zip(*_temp))
full_track = pd.DataFrame(columns=["Frame", "Box"])
for i in range(len(frames) - 1):
f1, f2 = frames[i], frames[i+1]
box1, box2 = np.array(track[i]), np.array(track[i+1])
#print(f1, box1)
#print(f2, box2)
full_track.loc[len(full_track)] = [f1, box1.tolist()]
#If there is a gap > 1 frame, interpolate
if f2 > f1 + 1:
#print("\t Gap, Between Frames ", f1, f2)
for f in range(f1+1, f2):
alpha = (f - f1) / (f2 - f1) # interpolation weight
box = ((1 - alpha) * box1 + alpha * box2).astype("int")
full_track.loc[len(full_track)] = [f, box.tolist()]
# Add the last box
full_track.loc[len(full_track)] = [frames[-1], track[-1]]
_temp = tracks_detected.query(f"ID == {ID}")[["Frame", "Confidence"]]
full_track = full_track.merge(_temp, how="left", on="Frame")
full_track["ID"] = ID
full_track["Confidence"] = full_track["Confidence"].ffill()
return full_track[["Frame", "ID", "Box", "Confidence"]]
2. Road Region Extraction
¶The camera view point is not fixed in our case, this open the neccesity to define a method to mask a road. The way to define the boundaries of the road region along the frames is based on:
- Applied Canny edge detection and Hough Line Transform to identify throad boundary lines.
- Incorporated constraints (expected line angles, approximate road height) and thresholding to filter out irrelevant lines.
- Created a trapezoidal mask representing the road region.
This process must be modified according to the video. With it is possible to transform perspective from camera view to bird's-eye view for further analysis of the tracked objects.
import sys
sys.path.append("/kaggle/input/auxfiles-track-road/")
from select_lines_road import calculate_lines, line_inter_x
from check_lines import sanity_check_lines, plot_road_lines, fit_points
def define_region(dir_lines):
# minimum limit of road to consider, parallel to inital road limit
x1,y1,x2,y2 = dir_lines["limit"]
lim_bellow = np.array([x1, y1+300, x2, y2+300], dtype="float")
# Intesection points
limit_function = fit_points(dir_lines["limit"], y_var=False)
left_function = fit_points(dir_lines["left"], y_var=False)
right_function = fit_points(dir_lines["right"], y_var=False)
cx_r = line_inter_x(dir_lines["limit"], dir_lines["right"]) ; cy_r = limit_function(cx_r)
cx_l = line_inter_x(dir_lines["limit"], dir_lines["left"]) ; cy_l = limit_function(cx_l)
lx_r = line_inter_x(lim_bellow, dir_lines["right"]) ; ly_r = right_function(lx_r)
lx_l = line_inter_x(lim_bellow, dir_lines["left"]) ; ly_l = left_function(lx_l)
#cv2.circle(fr, (int(cx_r), int(cy_r)), 15, (250,250,0), -1)
#cv2.circle(fr, (int(cx_l), int(cy_l)), 15, (250,0,250), -1)
pts = np.array([[int(lx_l), int(ly_l)], # bottom-left
[int(lx_r), int(ly_r)], # bottom-right
[int(cx_r), int(cy_r)], # top-right
[int(cx_l), int(cy_l)] # top-left
], np.int32)
return pts
def mask_road(frame, pts, sep_line=None):
fr = frame.copy()
# Fill trapezoid on overlay
if sep_line is None:
cv2.fillPoly(fr, [pts], color=(0, 255, 0)) # Green area
else:
sep_lx, sep_ly, sep_rx, sep_ry = sep_line
(lx_l,ly_l), (lx_r,ly_r), (cx_r,cy_r), (cx_l,cy_l) = list(pts)
midx_low = (lx_l + lx_r)/2; midy_low = (ly_l + ly_r)/2
midx_up = (cx_l + cx_r)/2; midy_up = (cy_l + cy_r)/2
sep_midx, sep_midy = (sep_lx+sep_rx)/2, (sep_ly+sep_ry)/2
left_poly = np.array([[lx_l, ly_l],
[midx_low,midy_low],
[sep_midx, sep_midy],
[sep_lx, sep_ly]], dtype=np.int32)
right_poly = np.array([[sep_midx, sep_midy],
[sep_rx, sep_ry],
[cx_r,cy_r],
[midx_up, midy_up]], dtype=np.int32)
cv2.fillPoly(fr, [left_poly], color=(0, 255, 0)) # Green area
cv2.fillPoly(fr, [right_poly], color=(0, 255, 0)) # Green area
# Blend overlay with original frame
alpha = 0.4 # transparency factor
fr = cv2.addWeighted(fr, alpha, frame, 1 - alpha, 0)
return fr
def mask_road_bird(frame):
h, w,_ = frame.shape
fr = frame.copy()
# Fill trapezoid on overlay
left_poly = np.array([[0,h],
[w/2,h],
[w/2,650],
[0, 650]], dtype=np.int32)
right_poly = np.array([[w/2, 650],
[w, 650],
[w,0],
[w/2,0]], dtype=np.int32)
cv2.fillPoly(fr, [left_poly], color=(0, 255, 0)) # Green area
cv2.fillPoly(fr, [right_poly], color=(0, 255, 0)) # Green area
# Blend overlay with original frame
alpha = 0.3 # transparency factor
fr = cv2.addWeighted(fr, alpha, frame, 1 - alpha, 0)
return fr
3. Bird’s-Eye View Transformation
¶A perspective transformation, to map the trapezoidal road region into a rectangular bird’s-eye view, is possible thanks to the assumption that:
- The highway trapezoid region on the camera view point is a rectangle on the bird's-eye view.
This simplifies vehicle motion analysis and distance measurement.
def transform_view(frame, src):
width, height = 400, 900 # adjust to your road dimensions
dst = np.array([
[0, height], # bottom-left
[width, height], # bottom-right
[width, 0], # top-right
[0, 0], # top-left
])
# Compute homography
M = cv2.getPerspectiveTransform(src.astype(np.float32), dst.astype(np.float32))
# Warp
warped = cv2.warpPerspective(frame, M, (width, height))
return warped
def transform_point(point, src):
width, height = 400, 900 # adjust to your road dimensions
dst = np.array([
[0, height], # bottom-left
[width, height], # bottom-right
[width, 0], # top-right
[0, 0], # top-left
])
# Compute homography
M = cv2.getPerspectiveTransform(src.astype(np.float32), dst.astype(np.float32))
return cv2.perspectiveTransform(point, M)
def transform_point_inv(src):
width, height = 400, 900 # adjust to your road dimensions
dst = np.array([
[0, height], # bottom-left
[width, height], # bottom-right
[width, 0], # top-right
[0, 0], # top-left
])
lp = np.float32([[[0,650]]])
rp = np.float32([[[width,650]]])
# Compute homography
M = cv2.getPerspectiveTransform( dst.astype(np.float32), src.astype(np.float32) )
lp = np.squeeze( cv2.perspectiveTransform(lp, M) )
rp = np.squeeze( cv2.perspectiveTransform(rp, M) )
return np.array( [lp[0], lp[1], rp[0], rp[1]] )
Vehicle Tracking and Road Detection¶
The vehicles boxes and the road boundaries region are store inside a pandas dataframe.
%%time
# Localize Road Lines and track Cars from Video
from IPython.display import display, clear_output
import pandas as pd
import numpy as np
cap = cv2.VideoCapture("/kaggle/input/traffic-videos/stable_small.mp4")
road_lines_track = pd.DataFrame(columns=["frame","left", "right", "limit","horizon"])
tracks_detected = pd.DataFrame([], columns=["Frame", "ID", "Box", "Confidence"])
###########################
time_s = 45 # seconds
fps = 30
###########################
i=0
while cap.isOpened():
ret, frame = cap.read()
if not ret: break # exit at final frame
#if i<30*30:
# i+=1
# continue
###################################### Track Cars
results = model.track(frame,
persist=True,
tracker="botsort.yaml", # better for crowded/overlapping scenes
classes=[2], # car only
conf=0.35,
iou=0.45,
imgsz=960)
data = results[0].boxes
if data.shape[0]!=0: # if cars were found
boxes = data.xyxy.cpu()
track_ids = data.id.int().cpu().tolist()
confidences = data.conf.cpu()
for box, track_id, conf in zip(boxes, track_ids, confidences):
box = tuple(map(int, box)) #x1,y1,x2,y2
tracks_detected.loc[len(tracks_detected)] = [i, track_id, box, conf.tolist()]
####################################
# Track Roads Lines
dir_lines = calculate_lines(frame) # #{"horizon": horizon, "limit": limit_line, "left":left_side, "right":right_side}
road_lines_track.loc[len(road_lines_track)] = [i, dir_lines["left"], dir_lines["right"],
dir_lines["limit"], dir_lines["horizon"]]
if i>fps*time_s: break
if i%5==0: clear_output(wait=True)
i+=1
# Cleanup
cap.release()
############# Smooth road lines
f_arr = lambda x: np.array(x, "float") if x is not None else x
# transform to array
for col in ["left","right","limit","horizon"]:
road_lines_track[col] = road_lines_track[col].apply(lambda x: f_arr(x))
# smooth lines variation
road_lines_track = sanity_check_lines(road_lines_track)
# Defining Road Region
road_lines_track["region"] = road_lines_track.apply(lambda row: define_region(row[["left","right","limit","horizon"]].to_dict()), axis=1)
road_lines_track["sep_line"] = road_lines_track["region"].apply(lambda x: transform_point_inv(x) )
############# Interpolate Tracks
tracks_interpolate = pd.DataFrame()
for _ID in tracks_detected["ID"].unique():
_full_track = interpolate_boxes(_ID, tracks_detected)
tracks_interpolate = pd.concat([tracks_interpolate,_full_track], ignore_index=True)
tracks_interpolate[["cx","cy"]] = pd.DataFrame(tracks_interpolate["Box"].apply(lambda t: ( (t[0]+t[2])/2, (t[1]+t[3])/2 ) ).tolist())
# Points in bird's-eye view
_new_points = []
for i, (_, _row) in enumerate(tracks_interpolate.iterrows()):
_row = _row.to_dict()
_frame = _row["Frame"]
_center_point = np.float32([[[_row["cx"],_row["cy"]]]])
_tr_points = road_lines_track.query(f"frame=={_frame}")["region"].iloc[0]
# Transform to bird's-eye view
point_warped = np.squeeze( transform_point(_center_point, _tr_points) )
_new_points.append(point_warped)
tracks_interpolate[["n_cx","n_cy"]] = pd.DataFrame(_new_points)
print("\nDone\n")
0: 544x960 9 cars, 176.8ms Speed: 4.5ms preprocess, 176.8ms inference, 3.0ms postprocess per image at shape (1, 3, 544, 960) Done CPU times: user 14min 9s, sys: 48 s, total: 14min 57s Wall time: 6min 7s
4. Vehicle Counting
¶- To count the vehicles, we set a reference horizontal line to separate the vehicles that already pass in both directions of the highway.
This reference line across the road can be set on the bird’s-eye view, then tranform it to the reference frame of the camera view. For visual purposes, we can fill the road region were the already counted vehicles are.
def take_frame(n_frame):
cap = cv2.VideoCapture("/kaggle/input/traffic-videos/stable_small.mp4")########################
i=0
while cap.isOpened():
ret, frame = cap.read()
if not ret: break
frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
if i==n_frame: break
i+=1
# Cleanup
cap.release()
return frame
n_frame = 100
frame = take_frame(n_frame=n_frame) #0, 170, 19, 87, 90
dir_lines = [v for k,v in road_lines_track.query(f"frame=={n_frame}").T.to_dict().items()][0]
fr = plot_road_lines(frame, dir_lines=dir_lines)
pts = dir_lines["region"]; sep = dir_lines["sep_line"]
fr2 = mask_road(frame, pts)
fr3 = mask_road(frame, pts, sep)
f,ax = plt.subplots(figsize=(10,6), ncols=2, nrows=2)
ax[0,0].imshow(frame) ; ax[0,1].imshow(fr)
ax[1,0].imshow(fr2) ; ax[1,1].imshow(fr3)
ax[0,0].set_title("Original Frame")
ax[0,1].set_title("Detected Road Lines")
ax[1,0].set_title("Detected Road")
ax[1,1].set_title("Detection Zones")
display(f)
plt.close()
5. Speed Estimation
¶To estimate velocity, by tracking vehicle displacement over time, can be done easily in the bird’s-eye view. The traveled distance is the y-axis displacement between frames. Using video frame rate, the displacement is transformed into speed.
We know that:
- 1 second pass every 30 frames
- Real highway size is undertermined, pixel units will be used.
# Cut Possible Video Distorsion Region tracks
_tracks = tracks_interpolate.query(f"n_cy >= {cut_bird}")
# ignore cars in side roads
tr = 100
exc_ids = _tracks.query(f"n_cx < -1*{tr} | n_cx > 400+{tr}")["ID"].unique()
# separate mistraked objects < 20 fps (less than a second)
id_group = _tracks.groupby("ID").size().reset_index()
id_group = id_group[id_group[0] >= 20]["ID"].values
r = []
for _id in id_group:
if _id not in exc_ids:
y_steps = _tracks.query(f"ID == {_id}")[["n_cy"]].sort_values("n_cy")
mean_velocity = (y_steps - y_steps.shift(1)).mean().iloc[0]*30
r.append([_id,mean_velocity])
r_df = pd.DataFrame(r, columns=["ID", "Velocity"]).sort_values("Velocity", ascending=False)
f,ax=plt.subplots()
r_df["Velocity"].hist(bins=20, ax=ax)
ax.set_title("Velocity Distribution")
ax.set_xlabel("Velocity (units per second)")
ax.set_ylabel("Count")
plt.show()
From the video we can infer that vehicles moves in a range of 180 and 320 units per second.
Result Visualization
¶An annotated video with the results found can be done using OpenCV (cv2.VideoWriter). Includes:
- Detected vehicles with bounding boxes
- Road mask and bird’s-eye view
- Vehicle counts
def combine_with_margins(fr, fr2):
# horizontal margin
black_bar = np.zeros((fr.shape[0], 40, 3), dtype=np.uint8)
combined = cv2.hconcat([black_bar, fr, black_bar]) # horizontal concat
combined = cv2.hconcat([combined, fr2, black_bar]) # horizontal concat
# vertical margin
black_bar = np.zeros((40, combined.shape[1], 3), dtype=np.uint8)
combined = cv2.vconcat([combined, black_bar]) # horizontal concat
#margin = 150 # pixels for title space
black_bar = np.zeros((150, combined.shape[1], 3), dtype=np.uint8)
combined = cv2.vconcat([black_bar, combined]) # horizontal concat
# --- Add titles with cv2.putText ---
cv2.putText(combined, "Camera View",
(500, 100), # position inside the top margin
cv2.FONT_HERSHEY_SIMPLEX, 2, (255, 255, 255), 2, cv2.LINE_AA)
cv2.putText(combined, "Bird's-eye View",
(1300, 100), # second title
cv2.FONT_HERSHEY_SIMPLEX, 2, (255, 255, 255), 2, cv2.LINE_AA)
return combined
cv2.putText(combined, "Bird's-eye View",
(1300, 100), # second title
cv2.FONT_HERSHEY_SIMPLEX, 2, (255, 255, 255), 2, cv2.LINE_AA)
return combined
def _contrast_text(frame, text, color, pos, size):
cv2.putText(frame, text, pos, cv2.FONT_HERSHEY_SIMPLEX, size, (0,0,0), int(10*size))
cv2.putText(frame, text, pos, cv2.FONT_HERSHEY_SIMPLEX, size, color, int(2*size))
cap = cv2.VideoCapture("/kaggle/input/traffic-videos/stable_small.mp4")
########################
# Get width & height from input video
w = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
h = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
# Define output video writer (MP4 with H.264-like codec)
fourcc = cv2.VideoWriter_fourcc(*'mp4v') # or 'XVID'
# video + birdview(720*400/900)
cut_bird = 200 # cut bird view to avoid deformations
w2 = int(h*400/(900-cut_bird))
out = cv2.VideoWriter("output.mp4", fourcc, 30, (w, h)) # fps=30
########################
car_counts = 0
crossed_ids = set()
########################
i=0
while cap.isOpened():
ret, frame = cap.read()
if not ret: break
###### Draw Road Lines
tr_pts, sep_line = road_lines_track.query(f"frame == {i}")[["region","sep_line"]].iloc[0]
fr = mask_road(frame, tr_pts, sep_line)
fr2 = transform_view(frame, tr_pts)
fr2 = mask_road_bird(fr2)
###### Draw Car Boxes
if tracks_interpolate.query(f"Frame == {i}").shape[0] != 0 : # There are no cars on the frame
data = tracks_interpolate.query(f"Frame == {i}")
for _, row in data.iterrows():
x1,y1,x2,y2 = row["Box"]; track_id = row["ID"]
# crop car from original frame, paste on blended image
car_crop = frame[y1:y2, x1:x2]
fr[y1:y2, x1:x2] = car_crop
# birds's-eye view points
xn, yn = map(int, [row["n_cx"], row["n_cy"]])
_contrast_text(fr2, f"ID: {track_id}",# {conf:.2f}",
(202,222,20), (xn-20, yn-20), size=1)
cv2.circle(fr2, (xn, yn), 10, (0,0,255), -1)
cv2.rectangle(fr, (x1,y1), (x2,y2), (202, 222, 20), 2)
# ID cars
_contrast_text(fr, f"ID: {track_id}",# {conf:.2f}",
(202,222,20), (x1, y1-10), size=1)
# check if the object has crossed the line
car_down = (yn>650) & (xn<200)
cat_up = (yn<650) & (xn>200)
if (car_down or cat_up):
# mark the object
cv2.rectangle(fr, (x1,y1), (x2,y2), (12, 226, 242), 2)
_contrast_text(fr, f"ID: {track_id}",# {conf:.2f}",
(50,250,255), (x1, y1-10), size=1)
# bird's eye view
_contrast_text(fr2, f"ID: {track_id}",# {conf:.2f}",
(50,250,255), (xn-20, yn-20), size=1)
if (track_id not in crossed_ids):
crossed_ids.add(track_id)
car_counts += 1
# Cars Coun Text
_contrast_text(fr, f"Cars: {car_counts}",
(50,250,0), (50, 50), size=1.25)
############
if i%5==0:
# Create figure
f = plt.figure(figsize=(10, 5)) # adjust total size
# Define a 1-row, 2-column grid, with relative widths [2,1]
gs = gridspec.GridSpec(1, 2, width_ratios=[4, 1])
ax1 = f.add_subplot(gs[0])
ax1.set_title(f"Camera View, Frame {i}")
ax1.imshow(cv2.cvtColor(fr, cv2.COLOR_BGR2RGB))
ax2 = f.add_subplot(gs[1])
ax2.set_title("Birds-eye view")
ax2.imshow(cv2.cvtColor(fr2, cv2.COLOR_BGR2RGB))
clear_output(wait=True)
display(f)
plt.close()
# Concatenate: original frame + plot
# Resize to match VideoWriter resolution
fr2 = fr2[cut_bird:,: ,:]
fr2 = cv2.resize(fr2, (w2, h))
combined = combine_with_margins(fr, fr2)
combined = cv2.resize(combined, (w, h))
i+=1
if i>fps*time_s: break
# Write to video
out.write(combined)
# Cleanup
cap.release()
out.release()
- Vehicles were detected and tracked robustly even with some video instability.
- Road region is extracted and transformed into a consistent bird’s-eye view.
- Cars are successfully counted in both directions.
- Approximate vehicle speeds are calculated.